Spotify Playlist Analysis with Python

Objective. Using Python, the goal of this project is to implement the k-means clustering algorithm, a technique often used in machine learning, and use it for data analysis. We write various functions using lists, sets, dictionaries, sorting, and graph data structures for computational problem-solving and analysis.

Part 1. Spotify API Data

Spotify is a popular audio streaming platform with an extensive music database. The Spotify API allows developers to access the platform’s data providing global insights into music listening habits around the world^[1]. Using the API requires an initial setup involving several steps. These steps include registering as a Spotify developer, creating an app, modifying the dashboard redirect URI, and storing the client ID and secret. After completing the initial steps for setup, we have access to the Spotify API and all its features.

Get Playlist Data from API

First, we create a Client Credentials Flow Manager used in server-to-server authentication by passing the necessary parameters to the Spotify OAuth class^[2]. We provide a client id and client secret to the constructor of this authorization flow, which does not require user interaction.

# Set client id and client secret
client_id = 'xxx'
client_secret = 'xxx'

# Spotify authentication
client_credentials_manager = SpotifyClientCredentials(client_id, client_secret)
sp = spotipy.Spotify(client_credentials_manager = client_credentials_manager)

Now we can get the full details of the tracks of a playlist based on a playlist ID, URI, or URL. Choose a specific playlist to analyze by copying the URL from the Spotify Player interface. Using that link, the following code uses the playlist_tracks method to retrieve a list of IDs and corresponding artists for each track from the playlist.

for link in playlist_links:
    playlist_URI = link.split("/")[-1].split("?")[0]
    # Iterate over list of tracks in playlist
    for i in sp.playlist_tracks(playlist_URI)["items"]:   
        track_ids.append(i['track']["id"]) # Extract song id
        artist_ids.append(i['track']["artists"][0]["uri"]) # Extract artist id

Then, we write a function that takes the playlist data from the API and gets the metadata and audio characteristics of each track. Specifically, the function reads the query results for a playlist and returns the track name, track ID, artist, album, duration, popularity, artist popularity, artist genre, and audio characteristics for each track.

name: The name of the track.
album: The name of the album on which the track appears.
artist: The name of the artist who performed the track.
release_date: The date the album was first released.
length: The track length in milliseconds.
popularity: The popularity of the track calculated by an algorithm based on the total number of plays the track has had and how recent those plays are.
artist_pop: The popularity of the artist calculated from the popularity of all the artist’s tracks.
artist_genres: A list of the genres the artist is associated with.

Spotify Audio Features

Spotify’s audio features are precalculated measures of both low-level and high-level perceptual music qualities that help classify a track. As indicated by the Spotify website, a quick explanation of each feature is shown below. More information on how to interpret these audio features is located at Spotify’s API documentation.

acousticness: A confidence measure of whether the track is acoustic.
danceability: Suitability for dancing based on tempo, rhythm, beat, and regularity.
energy: A perceptual measure of intensity and activity.
instrumentalness: Predicts whether a track contains no vocals.
liveness: Probability that the track was performed live.
loudness: Overall loudness of a track in decibels (dB).
speechiness: Detects the presence of spoken words in a track.
tempo: Estimated pace of a track in beats per minute (BPM).
valence: A measure describing the musical positiveness.

The following code loops through each track ID in the playlist and extracts the song information by calling the function we created. From there, we can create a dataframe by passing in the returned data using the pandas package.

# Loop over track ids
all_tracks = [playlist_features(track_ids[i], artist_ids[i], playlist_ids[i]) 
              for i in range(len(track_ids))]

name	album	artist	release_date	length	popularity	artist_pop	artist_genres	acousticness	danceability	energy	instrumentalness	liveness	loudness	speechiness	tempo	valence
2 AM	Pure Infinity	SwaVay	2019-05-24	198577	54	51	[‘atl hip hop’, ‘indie hip hop’, ‘underground hip hop’]	0.434	0.783	0.341	9.85e-05	0.362	-12.353	0.0727	126.799	0.184
Golden Child	Lady Wrangler	Shaboozey	2018-10-05	177773	45	56	[‘pop rap’]	0.362	0.792	0.591	1.90e-06	0.360	-8.848	0.2900	151.029	0.365

Part 2. Similar Artists

First, we want to find the most frequently occurring artist in a given playlist. We use the value_counts function to get a sequence containing counts of unique values sorted in descending order.

# Count distinct values in column
tallyArtists = df.value_counts(["artist", "artist_id"]).reset_index(name='counts')
topArtist = tallyArtists['artist_id'][1]

artist	artist_id	counts
Juice WRLD	4MCBfE4596Uoi2O4DtmEMz	10
Post Malone	246dkjvS1zLTtiykXe5h60	8
SAINt JHN	0H39MdGGX6dbnnQPt6NQkZ	3

I can retrieve artist and artist-related data using the following code, passing the artist ID to the artist and artist-related artist functions under the spotipy package. The returned list of similar artists is sorted by similarity score based on the listener data^[3].

a = sp.artist(topArtist)
ra = sp.artist_related_artists(topArtist)

Below is a sample of the result when we query Spotify for the most similar artists to the playlist’s top artist, creating a list that holds all of the artist source ids and target ids. We retrieve similar data for the nodes of the connection graph, creating a list that holds information for each specified artist.

source_name	source_id	target_name	target_id
Post Malone	246dkjvS1zLTtiykXe5h60	Rae Sremmurd	7iZtZyCzp3LItcw1wtPI3D
Post Malone	246dkjvS1zLTtiykXe5h60	Huncho Jack	6extd4B6hl8VTmnlhpl2bY
Post Malone	246dkjvS1zLTtiykXe5h60	Tyla Yaweh	1MXZ0hsGic96dWRDKwAwdr

Let’s see how things look when we pull in the full dataset, with each of the artist’s top most similar artists and each of their most similar artists. The following visualization is based on the Spotify Similiar Artists API article and created with flourish studio.

Part 3. Track Similarity Search

Objective. Design and implement a k-means clustering algorithm in Python.

K-means clustering is a popular machine learning and data mining algorithm that discovers possible clusters within a dataset. Finding these clusters often reveals meaningful information from the data distribution. Below, we create a query to retrieve similar elements based on the k-Nearest Neighbors (KNN) using the Euclidean distance.

Definitions

As with many machine learning techniques, this algorithm consists of a vast list of terminology which we define in a bit more detail below.

Definition 1. Distance The Euclidean distance, which indicates a straight line, is a simple way to calculate how close a data point is to a centroid using the Pythagorean theorem. For two points \(a = \left[a_1, a_2, \ldots, a_n\right]\) and \(b = \left[b_1, b_2, \ldots, b_n\right]\), where \(n\) is the current dimension, we define the euclidean distance between both points as

\[ \small \begin{align} \mathbf{\color{darkmagenta} D}(a, b) &= \sqrt{\left(a_1 - b_1\right)^2 + \left(a_2 - b_2\right)^2 + \ldots + \left(a_n - b_n\right)^2} \end{align} \]

Definition 2. Clusters A cluster is a collection of points that are part of the same group. For k-means, every point is part of a cluster. So as the algorithm progresses and the centroids shift, points might change which cluster they’re grouped in, even though the point itself does not move.

Definition 3. Centroids A centroid is the center of a cluster calculated by the average location of all the cluster points. This is equivalent to the average of the data points’ components in each dimension. So if we have three \(n\)-dimensional points \(a\), \(b\), and \(c\), we define the average as

\[ \small \mathrm{average} = \left[ \tfrac{a_1 + b_1 + c_1}{3}, \tfrac{a_2 + b_2 + c_2}{3}, \tfrac{a_3 + b_3 + c_3}{3} \right] \]

Definition 4. Convergence An algorithm converges if the locations of all centroids do not change much between two iterations, e.g. within some threshold of \(1 \times 10^{-5}\).

KNN Algorithm

The KNN algorithm^[4] searches for \(k\) similar elements based on a query point at the center within a predefined radius. The Euclidean distance between two points is the length of the line segment between the two points. In this sense, the closer the distance is to 0, the more similar the songs are.

K-means clustering works in four steps:

Initialize some number \(k\) of cluster centers, also called centroids.
For each data point in the dataset, assign it to the closest centroid.
Update the locations of the centroids to be the average of all the points assigned to that cluster.
Repeat steps 2 and 3 until convergence.

Note that the actual data points do not change. Only the locations of the centroids change with each iteration. And as the centroids move, the set containing the data points closest to each centroid alters.

KNN Query Example

Our function allows us to create personalized query points and modify the columns to explore other options. For example, the following code selects a specific set of song attributes and then searches for the \(k\) highest values of these attributes set equal to one. Let’s search for \(k=3\) similar songs to a query point \(\textrm{songIndex} = 6\).

# Select song and column attributes
query_point = 4
columns = ['acousticness','danceability','energy','instrumentalness','liveness','speechiness','valence']
# Set parameters and run query
func, param = knnQuery, 3
response = querySimilars(df, columns, query_point, func, param)

## ---- Query Point ----

## AG Club - Memphis

## ---- k = 3 similar songs ----

## Roddy Ricch - Stop Breathing
## The Game - Eazy
## Lil Yachty - Yacht Club (feat. Juice WRLD)

## ---- k = 3 nonsimilar songs ----

## Post Malone - Internet
## ODIE - In My Head
## Frank Ocean - In My Room

The code below implements the same idea as above, but queries each track in a given playlist instead of a single defined query point.

similar_count = {} # Similar songs count
nonsimilar_count = {} # Non-similar songs count
for track_index in df.index:
    response = querySimilars(df, columns, track_index, func, param)
    for similar_index in response[0]: # Get similar songs
        track = getMusicName(df.loc[similar_index])
        if track in similar_count:
            similar_count[track] += 1
        else:
            similar_count[track] = 1
    for nonsimilar_index in response[1]: # Get non-similar songs
        track = getMusicName(df.loc[nonsimilar_index])
        if track in nonsimilar_count:
            nonsimilar_count[track] += 1
        else:
            nonsimilar_count[track] = 1

## ---- NON-SIMILAR SONGS COUNT ----

## Frank Ocean - In My Room : 83
## ODIE - In My Head : 46
## Post Malone - Internet : 42
## Blxst - Hurt : 25
## Lil Uzi Vert - The Way Life Goes (feat. Nicki Minaj & Oh Wonder) - Remix : 19
## Kanye West - Waves : 12
## Tyla Yaweh - Understand Me : 11
## Mark Battles - Lemme Talk : 9
## Kanye West - Violent Crimes : 9
## SAINt JHN - The Best Part of Life : 8

Part 4. K Means Clustering

Next, we implement the K-Means clustering algorithm using the Scikit-Learn library to break down a playlist into several smaller playlists. The unsupervised learning algorithm divides similar data points into k groups by computing the distance to the centroid.

The first step is to define an appropriate predefined number (k) of clusters. We use the Elbow Method to determine the optimal k, as shown below^[5].

## C:\PROGRA~3\Anaconda3\lib\site-packages\sklearn\cluster\_kmeans.py:1036: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=1.
##   warnings.warn(

Thus, we tune the clustering algorithm by running K-Means for a range of k values, obtaining the above figure. It looks like a value of 3 is optimal for this case. Next, we call the K-Means function and set the k value to 3 clusters.

from sklearn.cluster import KMeans
model = KMeans(n_clusters = 3)
model = model.fit(features)
my_df['cluster'] = list(model.labels_)

Considering that there are seven different audio features for the clustering task, we use principal component analysis (PCA) to reduce the dimensionality of the data into a more easily visualized set of variables.

from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
pca_result = pca.fit_transform(features)

In the above code, we define a PCA instance to find two principal components determined from the features of the data. From there, we visualize the resulting clusters and explore the variation. The figure below shows our 3 clusters represented in 2-dimensional space.

	acousticness	danceability	energy	instrumentalness	liveness	speechiness	valence
1	0.1193033	0.6751	0.6478	0.0002468313	0.1572567	0.1539933	0.5820333
2	0.5396875	0.737625	0.5372187	0.02251566	0.1513125	0.1588813	0.4577406
3	0.1411991	0.6402273	0.5815455	0.0002416193	0.1652682	0.1366114	0.2303227

Cluster 1 has the highest energy and valence, indicating that these tracks are faster-paced, louder, and more positive (e.g., happy, cheerful, euphoric) than the other clusters.
Cluster 2 has the highest acousticness, with a mean value of 0.5396875 over all the cluster’s songs. Cluster 2 is also higher in danceability, indicating tracks with a faster tempo and beat intensity.
Cluster 3 appears to be the lowest valence, with a mean of 0.2303227, indicating more negative trajectories (e.g., sadness, frustration, anger).
All the clusters have values below 0.33, indicating that the songs most likely represent music and other non-speech-like tracks.